Explaining Preference Learning

Although there is existing work on learning user preferences in various systems, the outputs of such systems tend to be confusing to users. Studies of users interacting with such systems show that when the system incorrectly predicts a user's preference, the user may attempt to correct the views of the system (for instance, by randomly providing more training examples). In many cases, however, users lose patience with this approach, largely due to a misunderstanding of how the underlying system is using the data to make predictions. Ultimately, they may stop trusting the system entirely. Even when the system is correct, users view such outcomes as "magical" in some way, but are unable to understand why a particular answer is correct, or whether the system is likely to be helpful in the future.